MULTIPLE-CHOICE QUESTIONS

MedTutor: A Retrieval-Augmented LLM System for Case-Based Medical Education (2025.emnlp-demos)

Challenge:	Domain-adaptive pre-training (DAPT) is one approach for enabling LLMs to handle unseen knowledge.
Approach:	They propose to disentangle the answering process into three subtasks and evaluate the performance of each subtask.
Outcome:	The proposed model resolves the elicitation task that the base model struggled with but does not resolve other subtasks.

Copied to clipboard

Dongsuk Jang, Ziyao Shangguan, Kyle Tegtmeyer, Anurag Gupta, Jan T Czerminski, Sophie Chheang, Arman Cohan

Information Retrieval

Challenge:	Existing educational tools for medical residents are time-consuming and inconsistent.
Approach:	They propose a system that generates educational content and multiple-choice questions from clinical case reports and a pipeline that takes clinical case report input and produces targeted educational materials.
Outcome:	The system generates educational content and multiple-choice questions from clinical case reports and synergizes with local knowledge base to ensure it is foundationally sound and current.

HANS, are you clever? Clever Hans Effect Analysis of Neural Systems (2024.starsem-1)

Copied to clipboard

Leonardo Ranaldi, Fabio Zanzotto

Every Answer Matters: Evaluating Commonsense with Probabilistic Measures (2024.acl-long)

Challenge:	Large Language Models (LLMs) have been exhibiting outstanding abilities to reason around cognitive states, intentions, and reactions of all people involved, letting humans guide and comprehend day-to-day social interactions effectively.
Approach:	They propose to use multiple-choice questions (MCQ) benchmarks to assess LLMs' ability to reason around cognitive states, intentions, and reactions of all people involved to investigate their resilience abilities.
Outcome:	The proposed models exhibit exceptional abilities to reason around cognitive states, intentions, and reactions of all people involved, letting humans guide and comprehend day-to-day social interactions effectively.

Copied to clipboard

Qi Cheng, Michael Boratko, Pranay Kumar Yelugam, Tim O’Gorman, Nalini Singh, Andrew McCallum, Xiang Li

An Information-Theoretic Approach to Analyze NLP Classification Tasks (2024.acl-long)

Challenge:	Existing commonsense evaluations are often posed as multiple-choice questions, allowing models to exploit systematic biases.
Approach:	They propose a generative task that evaluates common sense via multiple open-ended generations and a method that strongly correlates with human judgments.
Outcome:	The proposed method outperforms strong language model baselines on a dataset of human and machine common sense.

Copied to clipboard

Luran Wang, Mark Gales, Vatsal Raina

Information Extraction & Text Mining

Challenge:	Natural language processing (NLP) tasks take either a single or multiple text elements to predict an output variable.
Approach:	They propose an information-theoretic framework to analyse the influence of inputs on the output of text classification tasks.
Outcome:	The proposed framework is available at: https://github.com/WangLuran/nlp-element-influence.

Exploring Fine-Tuning for In-Context Retrieval and Efficient KV-Caching in Long-Context Language Models (2026.eacl-short)

Copied to clipboard

Francesco Maria Molfese, Momchil Hardalov, Rexhina Blloshmi, Bill Byrne, Adrià de Gispert

Answer Uncertainty and Unanswerability in Multiple-Choice Machine Reading Comprehension (2022.findings-acl)

Challenge:	Long-Context Language Models (LCLMs) can encode entire document collections, offering a strong alternative to retrieval-augmented generation (RAG).
Approach:	They propose to use LCLMs to encode documents with context windows of millions of tokens to improve their performance.
Outcome:	The proposed training strategies improve long-context performance and their robustness under compression techniques.

Copied to clipboard

Vatsal Raina, Mark Gales

When Retriever-Reader Meets Scenario-Based Multiple-Choice Questions (2021.findings-emnlp)

Challenge:	Machine reading comprehension (MRC) systems focus on selecting the correct answer to a question given a context paragraph.
Approach:	They propose to use machine reading comprehension (MRC) to assess the ability of systems to understand natural language.
Outcome:	The proposed system outperforms a system built with an NOA option . the results show that the system is not confident about the NOA choice .

Copied to clipboard

ZiXian Huang, Ao Wu, Yulin Shen, Gong Cheng, Yuzhong Qu

Information Retrieval

Challenge:	Scenario-based question answering (SQA) requires retrieving and reading paragraphs from a large corpus to answer a question contextualized by a long scenario description.
Approach:	They propose a model where the retriever is implicitly supervised only using QA labels via a novel word weighting mechanism.
Outcome:	The proposed model outperforms strong baselines on multiple-choice questions in three datasets.

Anchored Answers: Unravelling Positional Bias in GPT-2’s Multiple-Choice Questions (2025.findings-acl)

Copied to clipboard

Ruizhe Li, Yanjun Gao

Large Language Models Sensitivity to The Order of Options in Multiple-Choice Questions (2024.findings-naacl)

Challenge:	Large Language Models (LLMs) exhibit a positional bias, particularly an even worse “anchored bias” in the GPT-2 family, where they consistently favour the first choice ‘A’ in MCQs.
Approach:	They propose to use the “logit lens” method to trace and modify the internal modules within GPT-2 models responsible for this bias.
Outcome:	The proposed approach mitigates the positional bias and improves the accuracy of the GPT-2 model across multiple datasets.

Copied to clipboard

Pouya Pezeshkpour, Estevam Hruschka

LongBench v2: Towards Deeper Understanding and Reasoning on Realistic Long-context Multitasks (2025.acl-long)

Challenge:	Large Language Models have demonstrated impressive capabilities in various NLP tasks, but previous studies have shown they are sensitive to prompt wording and few-shot demonstrations and their order.
Approach:	They focus on LLMs robustness on multiple-choice questions . they find a performance gap of 13% to 85% when options are reordered .
Outcome:	The proposed model outperforms supervised models on multiple choice questions even outperforming humans.

Copied to clipboard

Yushi Bai, Shangqing Tu, Jiajie Zhang, Hao Peng, Xiaozhi Wang, Xin Lv, Shulin Cao, Jiazheng Xu, Lei Hou, Yuxiao Dong, Jie Tang, Juanzi Li

DeSIQ: Towards an Unbiased, Challenging Benchmark for Social Intelligence Understanding (2023.emnlp-main)

Challenge:	Existing benchmarks on longcontext large language models fail to reflect their deep understanding capabilities across diverse tasks.
Approach:	They propose a benchmark to assess the ability of long-context large language models to handle long-text problems.
Outcome:	The proposed model achieves 50.1% accuracy when directly answering the questions . human experts achieve only 53.7% accuracy under a 15-minute time constraint .

Copied to clipboard

Xiao-Yu Guo, Yuan-Fang Li, Reza Haf

Challenge:	Social intelligence is essential for understanding and reasoning about human expressions, intents and interactions.
Approach:	They propose a methodology to study the soundness of Social-IQ by applying simple perturbations to a dataset of multiple choice questions on videos of complex social interactions.
Outcome:	The proposed method reduces biases in the original dataset and improves performance.

Multi2Claim: Generating Scientific Claims from Multi-Choice Questions for Scientific Fact-Checking (2023.eacl-main)

Copied to clipboard

Neset Tan, Trung Nguyen, Josh Bensemann, Alex Peng, Qiming Bao, Yang Chen, Mark Gahegan, Michael Witbrock

CBT-Bench: Evaluating Large Language Models on Assisting Cognitive Behavior Therapy (2025.naacl-long)

Challenge:	Existing scientific fact-checking datasets are limited due to expertise bottleneck . multi2Claim pipeline is a tool to convert multiple-choice questions into fact- checking data .
Approach:	They propose a pipeline for automatically converting multiple-choice questions into fact-checking data . they generate two large-scale datasets for scientific-fact-checker tasks . success at this task can help the reader understand scientific topics and promote science .
Outcome:	The proposed pipeline improves performance on two large-scale scientific fact-checking datasets.

Copied to clipboard

Mian Zhang, Xianjun Yang, Xinlu Zhang, Travis Labrum, Jamie C. Chiu, Shaun M. Eack, Fei Fang, William Yang Wang, Zhiyu Chen

FIBER: Fill-in-the-Blanks as a Challenging Video Understanding Evaluation Framework (2022.acl-long)

Challenge:	Existing research has explored mental health condition classifications, empathetic conversations, and chatbots designed for simple discourse structures.
Approach:	They propose a benchmark for systematic evaluation of cognitive behavioral therapy assistance using Large Language Models (LLMs).
Outcome:	The proposed benchmark includes three levels of tasks covering key aspects of cognitive behavioral therapy that could be enhanced through AI assistance.

Copied to clipboard

Santiago Castro, Ruoyao Wang, Pingxuan Huang, Ian Stewart, Oana Ignat, Nan Liu, Jonathan Stroud, Rada Mihalcea

Multimodality

Challenge:	Existing video understanding evaluation frameworks that use fill-in-the-blanks do not reflect real-world tasks.
Approach:	They propose to use fill-in-the-blanks as a video understanding evaluation framework and introduce a novel dataset that collects multiple perspectives on the same video.
Outcome:	The proposed framework does not share the weaknesses of the current state-of-the-art language-informed video understanding tasks, namely: (1) video question answering using multiple-choice questions, where models perform relatively well because they exploit linguistic biases in the task formulation; (2) video captioning, which relies on an open-ended evaluation framework that is often inaccurate because system answers may be perceived as incorrect if they differ in form from the ground truth.

This Land is Your, My Land: Evaluating Geopolitical Bias in Language Models through Territorial Disputes (2024.naacl-long)

Copied to clipboard

Bryan Li, Samar Haider, Chris Callison-Burch

Cosmos QA: Machine Reading Comprehension with Contextual Commonsense Reasoning (D19-1)

Challenge:	Pretrained large language models may answer differently in different languages . this contrasts with a multilingual human, who would likely answer consistently .
Approach:	They propose a dataset of territorial disputes which includes multiple-choice questions in 49 languages . they propose metrics to quantify bias and consistency in responses across different languages based on their data .
Outcome:	The proposed model recalls certain knowledge inconsistently when asked in different languages.

Copied to clipboard

Lifu Huang, Ronan Le Bras, Chandra Bhagavatula, Yejin Choi

Mitigating Selection Bias with Node Pruning and Auxiliary Options (2025.acl-long)

Challenge:	Existing reading comprehension datasets focus on factual and literal understanding of context paragraphs, but our dataset focuses on reading between the lines over a diverse collection of everyday narratives.
Approach:	They propose a large-scale dataset that requires commonsense-based reading comprehension, formulated as multiple-choice questions.
Outcome:	The proposed architecture improves over the baselines of existing reading comprehension datasets and shows a significant gap between machine (68.4%) and human performance (94%).

Copied to clipboard

Hyeong Kyu Choi, Weijie Xu, Chi Xue, Stephanie Eckman, Chandan K. Reddy

Challenge:	Large language models exhibit systematic preferences for answer choices when answering multiple-choice questions.
Approach:	They propose two methods to identify and remove internal sources of selection bias . they propose Choice Kullback-Leibler Divergence (CKLD) to capture distributional imbalances in model predictions.
Outcome:	The proposed methods improve answer accuracy while reducing selection bias.

Fake Alignment: Are LLMs Really Aligned Well? (2024.naacl-long)

Copied to clipboard

Yixu Wang, Yan Teng, Kexin Huang, Chengqi Lyu, Songyang Zhang, Wenwei Zhang, Xingjun Ma, Yu-Gang Jiang, Yu Qiao, Yingchun Wang

Pattern Recognition or Medical Knowledge? The Problem with Multiple-Choice Questions in Medicine (2025.acl-long)

Challenge:	Existing studies on large language models have shown that they are poorly aligned in practice.
Approach:	They propose a framework to evaluate safety in large language models . they propose two new metrics to quantify fake alignment and obtain corrected performance estimation.
Outcome:	The proposed framework and two metrics show that some models with purported safety are poorly aligned in practice.

Copied to clipboard

Maxime Griot, Jean Vanderdonckt, Demet Yuksel, Coralie Hemptinne

QRMeM: Unleash the Length Limitation through Question then Reflection Memory Mechanism (2024.findings-emnlp)

Challenge:	Large Language Models (LLMs) are often evaluated using multiple-choice questions (MCQs) modeled on exams like the USMLE.
Approach:	They created a fictional medical benchmark centered on an imaginary organ, the Glianorex, to separate memorized knowledge from reasoning ability.
Outcome:	The proposed model outperforms base models in English but not in French.

Copied to clipboard

Bo Wang, Heyan Huang, Yixin Cao, Jiahao Ying, Wei Tang, Chong Feng

Question Answering

Challenge:	Existing methods for processing large textual content face insufficient adaptation to task-specific needs and missing multi-segmentation relationships.
Approach:	They propose a question then reflection memory mechanism which integrates a dual-structured memory pool and a structured graph guidance to facilitate a reflective trial-and-error approach for navigating and identifying relevant segments.
Outcome:	The proposed model achieves superior performance on multiple-choice questions and multi-doc QA.

OLMES: A Standard for Language Model Evaluations (2025.findings-naacl)

Copied to clipboard

Yuling Gu, Oyvind Tafjord, Bailey Kuehl, Dany Haddad, Jesse Dodge, Hannaneh Hajishirzi

Differentiable Open-Ended Commonsense Reasoning (2021.naacl-main)

Challenge:	Existing models claim to perform better on tasks measuring model capabilities, but there is no standard setup for reproducible evaluations.
Approach:	They propose a document that is documented and practical for reproducible LLM evaluations and includes recommendations from existing literature and new experiments.
Outcome:	The proposed standard identifies and reviews the varying factors in evaluation practices adopted by the community, such as prompt formatting, choice of in-context examples, probability normalizations, and task formulation.

Copied to clipboard

Bill Yuchen Lin, Haitian Sun, Bhuwan Dhingra, Manzil Zaheer, Xiang Ren, William Cohen

Leveraging Large Language Models for Learning Complex Legal Concepts through Storytelling (2024.acl-long)

Challenge:	Existing commonsense reasoning models work by scoring a question-candidate pair, but new approaches are needed to answer multiple-choice questions.
Approach:	They propose to use a corpus of commonsense facts to answer a commonsensical question without any pre-defined choices as a resource.
Outcome:	The proposed model outperforms baseline methods by a large margin in the open-ended commonsense reasoning task.

Copied to clipboard

Hang Jiang, Xiajie Zhang, Robert Mahari, Daniel Kessler, Eric Ma, Tal August, Irene Li, Alex Pentland, Yoon Kim, Deb Roy, Jad Kabbara

CommonsenseQA: A Question Answering Challenge Targeting Commonsense Knowledge (N19-1)

Challenge:	a novel application of large language models (LLMs) to legal education helps non-experts learn complex legal concepts . authors find storytelling helps nonexperts understand complex legal terms and concepts compared to definitions .
Approach:	They propose a novel application of large language models to legal education . they use LLMs to generate legal stories explaining complex legal concepts .
Outcome:	The proposed method improves comprehension and interest among non-native speakers compared to definitions . the novel method also shows that non-experts retain more stories .

Copied to clipboard

Alon Talmor, Jonathan Herzig, Nicholas Lourie, Jonathan Berant

GreekMMLU: A Native-Sourced Multitask Benchmark for Evaluating Language Models in Greek (2026.findings-acl)

Challenge:	Recent work on question answering relies on factoid questions with little general knowledge.
Approach:	They propose a dataset to capture commonsense question answering with prior knowledge . they extract multiple-choice questions that discriminate between the source and target concepts .
Outcome:	The proposed dataset captures commonsense reasoning beyond associations . it obtains 56% accuracy, well below human performance, which is 89% .

Copied to clipboard

Yang Zhang, Mersin Konomi, Christos Xypolopoulos, Konstantinos Divriotis, Konstantinos Skianis, Giannis Nikolentzos, Giorgos Stamou, Guokan Shang, Michalis Vazirgiannis

B-REASO: A Multi-Level Multi-Faceted Bengali Evaluation Suite for Foundation Models (2025.findings-emnlp)

Challenge:	Existing evaluation benchmarks for large language models are limited for Greek . Existing datasets are often machine-translated from English, failing to capture Greek linguistic and cultural characteristics.
Approach:	They propose a native-sourced benchmark for massive multitask language understanding in Greek . they publicize 16,857 samples and reserve 4,948 samples for a private leaderboard .
Outcome:	The proposed model is based on 21,805 multiple-choice questions across 45 subject areas . the model is publicly released and reserved for a private leaderboard .

Copied to clipboard

Md Tanzib Hosain, Md Kishor Morol

DiVERT: Distractor Generation with Variational Errors Represented as Text for Math Multiple-choice Questions (2024.emnlp-main)

Challenge:	B-REASO is the first inclusive assessment suite for advanced foundation model knowledge and reasoning skills in a Bengali language setup.
Approach:	We provide a Bengali assessment suite to assess advanced foundation model knowledge and reasoning skills in a language setup.
Outcome:	The B-REASO includes multiple-choice questions with four different degrees of difficulty . the questions cover 50 different fields, from science and engineering to the humanities .

Copied to clipboard

Nigel Fernandez, Alexander Scarlatos, Wanyong Feng, Simon Woodhead, Andrew Lan

Increasing Probability Mass on Answer Choices Does Not Always Improve Accuracy (2023.emnlp-main)

Challenge:	a new variational approach to distractors in multiple-choice questions is needed . high-quality distractors are crucial to the assessment and pedagogical value of MCQs . a variational method that learns the error behind distractors is more effective .
Approach:	They propose a variational approach that learns an interpretable representation of errors behind distractors in math MCQs.
Outcome:	The proposed method outperforms state-of-the-art approaches on distractors in math MCQs.

Copied to clipboard

Sarah Wiegreffe, Matthew Finlayson, Oyvind Tafjord, Peter Clark, Ashish Sabharwal

Susu Box or Piggy Bank: Assessing Cultural Commonsense Knowledge between Ghana and the US (2024.emnlp-main)

Challenge:	Pretrained language models (LMs) are used to discriminate on multiple-choice tasks that place probability mass on vocabulary tokens that aren’t among the given answer choices.
Approach:	They propose a mathematical formalism for SFC which allows us to quantify and bound its impact for the first time.
Outcome:	The proposed method eliminates the impact of SFC in the majority of instances.

Copied to clipboard

Christabel Acquaye, Haozhe An, Rachel Rudinger

MCQG-SRefine: Multiple Choice Question Generation and Evaluation with Iterative Self-Critique, Correction, and Comparison Feedback (2025.naacl-long)

Challenge:	Recent work has highlighted the culturally-contingent nature of commonsense knowledge . a multi-stage process is used to evaluate the commonsence of English LLMs .
Approach:	They propose a test set of 525 multiple-choice questions to evaluate commonsense knowledge of English LLMs in Ghana and the u.s. They use existing commonsensible datasets to rewrite them in a multi-stage process.
Outcome:	The proposed model improves on the culturally-contingent commonsense knowledge of English LLMs in Ghana and the United States.

Copied to clipboard

Zonghai Yao, Aditya Parashar, Huixue Zhou, Won Seok Jang, null Feiyun Ouyang, Zhichao Yang, Hong Yu

Low-Resource NLP

Challenge:	Generating multiple-choice questions (MCQG) for professional exams is challenging due to outdated knowledge, hallucination issues, and prompt sensitivity.
Approach:	They propose a framework for converting medical cases into high-quality USMLE-style questions using a self-refine-based framework.
Outcome:	The proposed framework improves human expert satisfaction regarding quality and difficulty of medical questions.

DisGeM: Distractor Generation for Multiple Choice Questions with Span Masking (2024.findings-emnlp)

Copied to clipboard

Devrim Çavuşoğlu, Seçil Şen, Ulaş Sert

It’s Not Easy Being Wrong: Large Language Models Struggle with Process of Elimination Reasoning (2024.findings-acl)

Challenge:	Multiple-choice cloze tests are a prevalent form of assessment that evaluates students' comprehension and inference abilities.
Approach:	They propose a framework for distractor generation using readily available pre-trained language models . human evaluations confirm that their approach produces more effective distractors .
Outcome:	The proposed framework outperforms existing methods without training or fine-tuning human evaluations confirm it.

Copied to clipboard

Nishant Balepur, Shramay Palta, Rachel Rudinger

PerCul: A Story-Driven Cultural Evaluation of LLMs in Persian (2025.naacl-long)

Challenge:	Recent research aims to unlock the reasoning capabilities of large language models (LLMs) chain-of-thought (COT) prompting can help LLMs reason toward correct answers, but its efficacy in reasoning toward incorrect answers is unexplored.
Approach:	They propose a task where large language models reason toward incorrect answers using chain-of-thought prompting.
Outcome:	The proposed task underperforms the strategy of choosing the correct answer on commonsense and scientific reasoning datasets.

Copied to clipboard

Erfan Moosavi Monazzah, Vahid Rahimzadeh, Yadollah Yaghoobzadeh, Azadeh Shakery, Mohammad Taher Pilehvar

Challenge:	Large language models predominantly reflect Western cultures due to the dominance of English-centric training data.
Approach:	They propose a dataset to assess the sensitivity of LLMs to Persian culture.
Outcome:	The proposed model shows a 11.3% gap between best closed-source model and layperson baseline while the gap increases to 21.3% by using the best open-weight model.

Enhancing Distractor Generation for Multiple-Choice Questions with Retrieval Augmented Pretraining and Knowledge Graph Integration (2024.findings-acl)

Copied to clipboard

Han Cheng Yu, Yu An Shih, Kin Man Law, KaiYu Hsieh, Yu Chen Cheng, Hsin Chih Ho, Zih An Lin, Wen-Chuan Hsu, Yao-Chung Fan

Unmasking Deceptive Visuals: Benchmarking Multimodal Large Language Models on Misleading Chart Question Answering (2025.emnlp-main)

Challenge:	Existing LMs undergo task-agnostic pertaining, but task-specific pretraining has gained prominence.
Approach:	They propose retrieval augmented pretraining and task-specific pretraining for DG . they propose to refine language model pretraining to align it more closely with downstream task .
Outcome:	The proposed method improves the performance of multiple-choice questions by integrating knowledge graphs and language models.

Copied to clipboard

Zixin Chen, Sicheng Song, KaShun Shum, Yanna Lin, Rui Sheng, Weiqi Wang, Huamin Qu

Multimodality

Challenge:	Misleading visualizations can distort perception and lead to incorrect conclusions.
Approach:	They propose a large-scale multimodal dataset to evaluate MLLMs on misleading chart reasoning.
Outcome:	The proposed framework evaluates MLLMs on misleading chart reasoning on a large-scale multimodal dataset spanning 21 misleader types and 10 chart types . it contains 3,026 curated examples spanning standard chart code, CSV data, multiple-choice questions, and labeled explanations, validated through iterative MLML checks and exhausted expert human review.

StoryAnalogy: Deriving Story-level Analogies from Large Language Models to Unlock Analogical Understanding (2023.emnlp-main)

Copied to clipboard

Cheng Jiayang, Lin Qiu, Tsz Chan, Tianqing Fang, Weiqi Wang, Chunkit Chan, Dongyu Ru, Qipeng Guo, Hongming Zhang, Yangqiu Song, Yue Zhang, Zheng Zhang

Can Model Uncertainty Function as a Proxy for Multiple-Choice Question Item Difficulty? (2025.coling-main)

Challenge:	Analogy-making between narratives is crucial for human reasoning . despite its importance, there has been limited research on story analogies .
Approach:	They construct a large-scale story-level analogy corpus with 24K story pairs . they find that the tasks are incredibly difficult for large language models such as ChatGPT .
Outcome:	The proposed corpus contains 24K story pairs from diverse domains with human annotations on two similarities from the extended Structure-Mapping Theory.

Copied to clipboard

Leonidas Zotos, Hedderik van Rijn, Malvina Nissim

Challenge:	Supervised approaches to difficulty estimation have yielded mixed results . generative large models are seen as a weakness when answering questions .
Approach:	They exploit generative large models to explore correlations between two different metrics of uncertainty, and the actual student response distribution.
Outcome:	The proposed model uncertainty is different in the case of correct vs wrong answers and the student response distribution is different.

DyePack: Provably Flagging Test Set Contamination in LLMs Using Backdoors (2025.emnlp-main)

Copied to clipboard

Yize Cheng, Wenxiao Wang, Mazda Moayeri, Soheil Feizi

Are Large Language Models Chronically Online Surfers? A Dataset for Chinese Internet Meme Explanation (2025.emnlp-main)

Challenge:	Open benchmarks are essential for evaluating large language models, but their accessibility makes them likely targets of test set contamination.
Approach:	They propose a framework that leverages backdoor attacks to flag models that used benchmark test sets during training.
Outcome:	The proposed framework detects models that trained on benchmark test sets without loss of logits or internal details . it can prevent false accusations while providing strong evidence for every detected case of contamination.

Copied to clipboard

Yubo Xie, Chenkai Wang, Zongyang Ma, Fahui Miao

LHMKE: A Large-scale Holistic Multi-subject Knowledge Evaluation Benchmark for Chinese Large Language Models (2024.lrec-main)

Challenge:	Large language models (LLMs) are trained on vast amounts of text from the Internet, but do they understand the viral content that rapidly spreads online?
Approach:	They introduce a dataset for CHinese Internet Meme Explanation that includes popular phrase-based memes from the Chinese Internet.
Outcome:	The proposed dataset includes popular phrase-based memes from the Chinese Internet, annotated with detailed information on their meaning, origin, example sentences, types, etc.

Copied to clipboard

Chuang Liu, Renren Jin, Yuqi Ren, Deyi Xiong

LexGenius: An Expert-Level Benchmark for Large Language Models in Legal General Intelligence (2026.findings-acl)

Challenge:	Existing benchmarks for comprehensively evaluating Chinese Large Language Models are insufficient.
Approach:	They propose a Large-scale, Holistic, and Multi-subject Knowledge Evaluation benchmark to evaluate Chinese Large Language Models.
Outcome:	The proposed benchmark measures the knowledge acquisition capabilities of Chinese Large Language Models across 75 subjects from primary school to professional certification exams.

Copied to clipboard

Wenjin Liu, Haoran Luo, Xin Feng, Xiang Ji, Lijuan Zhou, Rui Mao, Jiapu Wang, Shirui Pan, Erik Cambria

Tailoring Diagnostic Modeling to Individual Learners: Personalized Distractor Generation via MCTS-Guided Reasoning Reconstruction (2026.acl-long)

Challenge:	Existing benchmarks for legal general intelligence (GI) are result-oriented and do not evaluate the legal intelligence of large language models (LLMs).
Approach:	They propose a Chinese legal benchmark for evaluating legal GI in large language models . they use recent legal cases and exam questions to create multiple-choice questions .
Outcome:	The proposed benchmarks lack a systematic evaluation of the legal intelligence of large language models (LLMs) the results show that even the best LLMs lagging behind human legal professionals.

Copied to clipboard

Tao Wu, Jingyuan Chen, Wang Lin, Jian Zhan, Mengze Li, Fangzhou Jin, Min Zhang, Kun Kuang, Fei Wu

Natural Language Interfaces

Challenge:	Current distractor generation methods produce shared distractors for all students, ignoring individual variations in reasoning, which limits their diagnostic effectiveness.
Approach:	They propose a method which tailors distractors to each student’s specific cognitive flaws, inferred from their past question-answering (QA) history.
Outcome:	The proposed framework outperforms existing methods in generating plausible distractors and adapts to group-level settings.

LC-Eval: A Bilingual Multi-Task Evaluation Benchmark for Long-Context Understanding (2025.findings-emnlp)

Copied to clipboard

Sheikh Jubair, Arwa Omayrah, Amal Alshammari, Alhanoof Althnian, Abdulhamed Alothaimen, Norah A. Alzahrani, Shahad D. Alzaidi, Nora Al-Twairesh, Abdulmohsen Al-Thubaity

Generating Plausible Distractors for Multiple-Choice Questions via Student Choice Prediction (2025.acl-long)

Challenge:	Recent advances in Large Language Models (LLMs) have demonstrated sophisticated capabilities, including the ability to process and comprehend extended contexts.
Approach:	They propose a bilingual, multi-task evaluation benchmark designed to evaluate long-context understanding in English and Arabic.
Outcome:	The proposed benchmark targets context lengths ranging from 4k to over 128k tokens.

Copied to clipboard

Yooseop Lee, Suin Kim, Yohan Jo

Challenge:	Multiple-choice questions (MCQs) are critical for identifying misconceptions and gaps in knowledge and accurately assessing students' understanding.
Approach:	They propose to train a model to generate distractors that are more likely to be selected by students by a pairwise ranker and a distractor generator via Direct Preference Optimization.
Outcome:	The proposed model outperforms baseline models and performs comparable to humans in various metrics including pairwise rank accuracy and distractor plausibility.

MIBench: Evaluating Multimodal Large Language Models over Multiple Images (2024.emnlp-main)

Copied to clipboard

Haowei Liu, Xi Zhang, Haiyang Xu, Yaya Shi, Chaoya Jiang, Ming Yan, Ji Zhang, Fei Huang, Chunfeng Yuan, Bing Li, Weiming Hu

Large Language Models Still Exhibit Bias in Long Text (2025.findings-acl)

Challenge:	Existing benchmarks and MLLMs focus on single-image input scenarios, leaving performance of ML models when handling multiple images underexplored.
Approach:	They propose a benchmark to evaluate fine-grained abilities of multimodal large language models in multi-image scenarios.
Outcome:	The proposed benchmark categorizes the multi-image abilities into three scenarios: MII, MKS and MIC.

Copied to clipboard

Wonje Jeung, Dongjae Jeon, Ashkan Yousefpour, Jonghyun Choi

CLARity: Reasoning Consistency Alone Can Teach Reinforced Experts (2026.acl-long)

Challenge:	Existing fairness benchmarks for large language models focus on simple tasks . a new framework evaluates biases in LLMs through essay-style prompts .
Approach:	They propose a framework that evaluates biases in large language models through essay-style prompts.
Outcome:	The proposed framework uncovers subtle biases difficult to detect in simple responses.

Copied to clipboard

Jiuheng Lin, Cong Jiang, Zirui Wu, Jiarui Sun, Yansong Feng

UrbanVideo-Bench: Benchmarking Vision-Language Models on Embodied Intelligence with Video Data in Urban Spaces (2025.acl-long)

Challenge:	Existing solutions to supervise the reasoning process are prohibitively expensive.
Approach:	They propose a cost-effective reinforcement learning framework that enhances reasoning quality using a small, general-purpose LLM only.
Outcome:	Experiments show that CLARity improves reasoning quality by 16.5% over standard outcome-based reinforcement learning (RL) human evaluations confirm substantial gains in factual correctness and reasoning coherence, leading to more trustworthy model outputs.

Copied to clipboard

Baining Zhao, Jianjie Fang, Zichao Dai, Ziyou Wang, Jirong Zha, Weichen Zhang, Chen Gao, Yue Wang, Jinqiang Cui, Xinlei Chen, Yong Li

Multimodality

Challenge:	Large multimodal models exhibit remarkable intelligence, yet their embodied cognitive abilities during motion in open-ended urban aerial spaces remain to be explored.
Approach:	They propose a benchmark to evaluate whether large multimodal models can process continuous first-person visual observations like humans.
Outcome:	The proposed model can process first-person visual observations like humans, enabling recall, perception, reasoning, and navigation.

Africa Health Check: Probing Cultural Bias in Medical LLMs (2025.emnlp-main)

Copied to clipboard

Charles Nimo, Shuheng Liu, Irfan Essa, Michael L. Best